Installing BigDFT From Source

Since BigDFT is made up of a suite of packages (see suite description), we have provided an installer script which manages dependencies and passing configuration options to the build systems of each different package. As already mentioned, this Installer.py script is based on the Jhbuild package.

Such an installer require a recipe file, which contains the configure line instructions to be applied to each of the packages. Depending from the compiler and architecture, different compilation instructions can be given to the various libraries which belong to the program suite. Of course, the source can be previously downloaded from the git repository of the suite.

A standard compilation might look something like this:

git clone https://gitlab.com/l_sim/bigdft-suite.git
mkdir build
cd build
python ../bigdft-suite/Installer.py autogen
python ../bigdft-suite/Installer.py \
       -f ubuntu_OMP.rc build

The above example compiles the whole suite according to the instructions indicated in the rcfiles/ubuntu_OMP.rc file distributed with the progam suite.

Such instructions can be also performed on a supercomputer frontend. In case git clone is not possible on such a frontend (some supercomputers do not have internet access), we advise to clone the repository locally by only exporting the latest commit, and then rsync the copy to the remote directory:

git clone --depth 1 https://gitlab.com/l_sim/bigdft-suite.git
rsync -auv bigdft-suite <remote_dir_URL>

Usage of the `Installer.py` script

There are various actions available in the Installer.py script in addition to the build command. To know them, just type

$ ../Installer.py help
Parsing arguments with method argparse...
Quick overview of the BigDFT suite Installer program
--------------------------------------------------
USAGE: Installer.py <action> <package>
--------------------------------------------------
Available actions:
autogen :
	 Perform the autogen in the modules which need that. For developers only.
build :
	 Compile and install the code with the given configuration.
buildone :
	 Build a single module of the suite
check :
	 Perform check in the bigdft branches, skip external libraries.
clean :
	 Clean the branches for a fresh reinstall.
cleanone :
	 Clean a single module of the suite
dist :
	 Creates a tarfile for the suite tailored to reproduce the compilation options specified.
dry_run :
	 Visualize the list of modules that will be compiled with the provided configuration in the 'buildprocedure.png' file.
link :
	 Show the linking line that have to be used to connect an external executable to the package (when applicable)
make :
	 Recompile the bigdft internal branches, skip configuring step.
startover :
	 Wipe out all the build directories and recompile the important parts
update :
	 Useful to update a pre-compiled branch after a merge
--------------------------------------------------
Available packages: ['futile', 'atlab', 'chess', 'liborbs', 'psolver', 'bigdft', 'PyBigDFT', 'spred', 'bigdft-client']
--------------------------------------------------
QIFI-QIFI-QIFI-QIFI-QIFI-QIFI-QIFI-QIFI-QIFI-QIFI- (Quick Instructions For the Impatient)
Ideally, there are two different policies:
Developer: From a development branch, start by "autogen", then "build"
     User: From a tarball, start by "build"
Perform the "dry_run" command to have a graphical overview of the building procedure

So for example, to build the tarball of bigdft-suite out of a valid branch, you may type

Installer.py dist bigdft

You should eventually have a tarfile named bigdft-suite.tar.gz in the build tree. Such tarfile may be used to build the suite in another machine.

Few important informations:

This script is intended to provide the end-user with a functional set of executables coming from the suite packages. It has to be used to (re-)work on the entire suite, not a single package only.
It does not replace the traditional make commands. A developer of a single package, say futile for example, should first compile and install (make install) the package in its build directory, then rebuild the dependencies. In such a case, it is advised to run Installer.py clean first, then the build command would work.
In a future version of the build, it is foreseen to merge the functionalities of the Installer.py and the custom bundler/jhbuild.py scripts.

Below we will go into detail about the installer script and the rc files used to configure the code for your desired platform.

Using the configuration file (`rcfile`)

The default behaviour (no information)

Working with a configuration file is the default behaviour of jhbuild. Therefore it is normally expected to provide a file to the Installer.py script (see next section on how to do that). If no files is provided in the command line, the script search for (priority order):

A file named buildrc in the current build directory;
A file in the directory rcfiles/ of the source tree that contains the hostname string (or a part if it) in its basename. If multiple files satisfy this condition, it offers a choice.

Therefore when asked to proceed for the installation, it is advised to pay attention in the initial message which is the file chosen for the configuration.

How to invoke a configuration file

The name of the configuration file might be specified with the -f option of the installer. The file might be either specified via its absolute path, or by its name. In this case, it is searched for in (priority order):

The current working directory;
The directory rcfiles/ of the source tree.

Manipulating the configuration file

The principle of the configuration file is to provide configure options which are different for each of the packaged of the suite (called modules in the jhbuild spirit). Therefore, to each of the package one must associate a configure line. Such information is provided in the dictionary module_autogenargs of jhbuild, and might be specified as follows (see e.g. the file dynamic.rc):

module_autogenargs['libxc'] = "CC=gcc --enable-shared"

or, alternatively:

module_autogenargs.update({'libxc': "CC=gcc --enable-shared"})

depending on your taste. If the BigDFT suite is compiled from a configure line, at the end of the first compilation a buildrc file is produced, that can be then used in the following compilations.

The Python syntax of the configuration file

An rcfile is invoked inside the collection of python modules provided by jhbuild. Therefore within this file the python syntax is necessary. This adds extra features that would otherwise be difficult to implement. For example, one might define functions that indicate common configuration options, or that retrieve the current working directory to define more elaborated configuation lines. For example, in a rcfile we may found (see for example the file mira.rc of the distribution):

def fcflags_short():
 return """'FCFLAGS= -g -O3'"""
[...]
module_autogenargs.update({
  'libxc': fcflags_short(),
 [...]
 }

We here prefer to use return values of functions rather that to define extra variables in the script as jhbuild warns if unknown variable names are found in the configuration file.

There are some practical examples of compilations on different architectures in the rcfiles directory of the package suite where you might find useful information on how the code has been compiled on different platforms and for different options.

Separating the build of the core packages from the upstream (useful for developers)

A two stage installation procedure is also available. In the first step, all the upstream dependencies are installed. In the second, the unique BigDFT sources are installed. This can be useful for development, where frequent (re-)compilation might be required. In order to do so, the first installation to be preformed would be the upstream-suite package. Notice that in this case, the bundler/jhbuild.py python script have to be employed.

mkdir <upstream_build> && cd <upstream_build>
python <sourcedir>/bundler/jhbuild.py \
       -f <path_to_rcfile> build upstream-suite

Then you add extra_prefixes=["<upstream_build>"] to the rcfile you wish to use for building the BigDFT sources. Once this is done, the bigdft core packages can be compiled, including the no_upstream condition:

mkdir <core_build> && cd <core_build>
python <sourcedir>/Installer.py \
       -f <rcfile> \
       build -a no_upstream

Installing from a `configure` line

As written above, packages were already configured and compiled with a previous BigDFT version. Therefore we have prepared a compilation method to compile the 1.8.x build system from a configure line.

The principle is to execute, in a build directory different of the source one, instead of the configure script, the following command:

<path_to_sources>/Installer.py build -c

Let us consider the example of the pilipili machine (internal L_Sim lab machine). Clearly, environment modules still have to be loaded:

module load intel/13.0.1 impi/4.1.0.024

Then the installer script can be used with the following command:

<path_to_sources>/Installer.py build -c 'FCFLAGS=-O2 -openmp' \
'--with-ext-linalg=-L/opt/intel/composer_xe_2013.1.117/mkl/lib/intel64 -lmkl_rt -lmkl_scalapack_lp64 -lmkl_blacs_openmpi_lp64 -liomp5 -lm' \
'--enable-opencl' 'CC=mpicc' \
CFLAGS='-openmp' 'CXX=mpicxx' 'FC=mpifc' 'F77=mpif77' 'FCLIBS= '

The following message dialog will appear:

Configuration chosen for the Installer:\
 Hostname: pilipili
 Source directory: /home/athelas/genovese/work/BigDFT/1.8\
 Compiling from a branch: True\
 Build directory: /local/genovese/binaries/1.8-ocl\
 Action chosen: build\
 Verbose: False\
 Jhbuild baseline: <path_to_sources>/jhbuild.py \
 Configuration options:\
   Source: Environment variable 'BIGDFT_CONFIGURE_FLAGS'\
   Value: '"FCFLAGS=-O2 -openmp" "--with-ext-linalg=-L/opt/intel/composer_xe_2013.1.117/mkl/lib/intel64 -lmkl_rt -lmkl_scalapack_lp64 -lmkl_blacs_openmpi_lp64 -liomp5 -lm" "--enable-opencl" "CC=mpicc" "CXX=mpicxx" "FC=mpifc" "F77=mpif77" "FCLIBS= " '\
Do you want to continue (Y/n)?

The Installer script has detected the different compilation options. It has filled the environment variable BIGDFT_CONFIGURE_FLAGS with the options passed after the -c option in the command line. By typing Y the bigdft bundle will build.

As we did not specified the -v option (type ./Installer.py help for the available commands and options), the code will be built in silent mode (this would correspond to the tinderbox option of JhBuild. You should have the following information in the output:

List of modules to be treated: ['libyaml', 'futile', 'psolver', 'libxc', 'libABINIT', 'GaIn', 'bigdft', 'spred']
libyaml : ['checkout', 'configure', 'build', 'install']
futile : ['checkout', 'configure', 'build', 'install']
psolver : ['checkout', 'configure', 'build', 'install']
libxc : ['checkout', 'configure', 'build', 'install']
libABINIT : ['checkout', 'configure', 'build', 'install']
GaIn : ['checkout', 'configure', 'build', 'install']
bigdft : ['checkout', 'configure', 'build', 'install']
spred : ['checkout', 'configure', 'build', 'install']

Then in the directory named buildlogs of the build tree you might find the index.html file that contains the status of the compilation.

The environment variable BIGDFT_CONFIGURE_FLAGS is a way to indicate some general options for all modules. A more powerful method is to use a rcfile configuration file which indicates all possible option for each module. Some examples are provided in the directory rcfiles. For advanced features, the different possible options are detailed in the corresponding documentation of the jhbuild package.

At the end of a successful compilation, you find in the build directory a file named buildrc that may be used for future compilation and to specify more flexible configurations options. See the rcfile section for details.

Linking external software with BigDFT-suite libraries

Since version 1.8.0 the build system of BigDFT is “generic” in the sense that it does not only allow the compilation of the main BigDFT software, but also of various sub-packages. This is useful if one is only interested in some of the packages distributed with BigDFT.

As an example we will show the compilation of the CheSS package, which itself depends on futile. It can be downloaded here: 2 After downloading the tar.gz execute the following steps:

tar -xzvf CheSS-0.1.1.tar.gz
cd CheSS-0.1.1
mkdir Build
cd Build
../Installer.py build chess -d -c FC=``\ \ `` CC=``\ \ `` FCFLAGS=``\ \ `` --with-ext-linalg=``\

A dialogue similar to this one should appear:

Configuration chosen for the Installer:
  Hostname: stephan-Latitude-E7450
  Source directory: /home/stephan/Documents/BigDFT/stablebranch
  Compiling from a branch: True
  Build directory: /home/stephan/Documents/BigDFT/stablebranch/Build-gnu_debug
  Action chosen: dist
  Verbose: True
  Jhbuild baseline: ../jhbuild.py -f buildrc
  Configuration options:
    Source: Configuration file '/home/stephan/Documents/BigDFT/stablebranch/Build-gnu_debug/buildrc'
Do you want to continue (Y/n)?

Confirm and wait until the compilation in complete. In order to link now another software with CheSS, run the command:

../Installer.py link chess

which should give you an output similar to this one:

--------- Linking line to build with package "chess":
-I/home/stephan/Downloads/CheSS-0.1.1/Build/install/include-L/home/stephan/Downloads/CheSS-0.1.1/Build/install/lib -lCheSS-1 -lfutile-1 -lblacs-openmpi -lblacsF77init-openmpi -llapack -lblas -lyaml -lrt -lfutile-1 -lblacs-openmpi -lblacsF77init-openmpi -llapack -lblas -lyaml -lrt
--------------------------------------------------

link with CheSS.

The BigDFT Installer class

class Installer.BigDFTInstaller(action, package, rcfile, conditions, verbose, quiet, yes)[source]

Class for the installation of the bigdft-suite.

Attributes:

action (str): Action to be performed.

package (str): Package to install.

rcfile (str): Configuration file.

conditions (list(str)): List of conditions.

verbose (bool): If True, more verbose.

quiet (bool): If True, no messages.

yes (bool): If True, ask a question.